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Abstract 

Cathy Sandeen is Vice President for Education Attainment and Innovation at 
American Council on Education, the nation’s largest organization providing 
leadership and advocacy for all sectors of higher education. Within this role 
she develops and articulates ACE’s attainment agenda and currently oversees a 
large research project on Massive Open Online Courses. Until January 2013, she 
served as dean of Continuing Education and UCLA Extension at the University 
of California, Los Angeles. This essay provides a brief primer on the evolution of 
MOOCs, an overview of major forces and trends shaping this evolution, and the 

role of assessment within the MOOC context. 
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Assessment’s Place in the New MOOC World 


f f hen Massive Open Online Courses, or MOOCs, propelled into our awareness 
in the summer of 2012, they were either hailed as the solution to closing the postsecondary 
attainment gap in the U.S. or denounced as an extremely disruptive technology that would 
change higher education as we know it. Now, a year later, we realize the truth is probably 
somewhere in between. MOOCs certainly attracted the attention of the higher education 
community, they fostered a great deal of innovation, experimentation, discussion and 
debate, and they gave us a vision of how we might scale education with quality. For many 
people they also appear to have legitimized online teaching and learning, an educational 
practice that has existed for fifteen years or more. 

Of course, the dynamics of developing a business model to finance MOOCs in 
a sustainable way and integrating this format into traditional degree programs are still 
evolving. One of the most promising aspects of MOOCs is that assessment of student 
learning has become central to any conversation. In this new MOOC world, assessment 
is not an “after-the-fact add on,” but must be fully considered and integrated from the 
beginning of course design. In this way, MOOCs provide an opportunity for the assessment 
community to move to the center in one of the most exciting and potentially important 
conversations emerging in higher education today. 

Description and History 

In the online education world, the acronym, “MOOC,” is not that new. It was 
coined in 2008 for an online course in “Connectivism and Connective Knowledge,” 
offered by the University of Manitoba. Following the spirit of the open courseware 
movement, the university also opened up the course to online “auditors,” students who 
joined the course for free. Unexpectedly, over 2,000 additional students enrolled on this 
basis. A movement was born (“Defining a MOOC,” 2013). 
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Between 2008 and 2011 a number of institutions experimented with the MOOC 
concept continuing in the open educational resources vein. These courses are based on open 
resources, are free to students, have no requirements to enroll, have no enrollment limits, 
have relatively low levels of faculty facilitation, and encouragement community formation- 
but offer no academic credit. Many of these early experimental courses were developed by 
Canadian institutions. Not all courses achieved the extraordinarily high enrollments we 
see today, but many of the other components of and practices within contemporary MOOCs 
evolved during this time. The first recorded U.S. MOOC appeared in 2011, a course called 
“Online Learning Today and Tomorrow,” with over 2,500 students, offered by the University 
of Illinois Springfield (“Defining a MOOC,” 2013). 


The enthusiasm behind the 
development of MOOC 
platforms is closely linked 
to the personal experiences 
of the founders. 


Open online education providers such as the Khan Academy, TED, and iTunesU 
also emerged during this time. These providers offered high-quality educationally-oriented 
video content that attracted large numbers of viewers. The content was rarely organized into 
full courses and did not offer academic credit. Content offered by these organizations could 
be considered supplementary to formal coursework, such as the tutorials offered by Khan 
Academy. These sources also tended to appeal to individuals seeking general knowledge or 
enrichment rather than progress toward a degree or credential. 


MOOCs entered the popular vernacular in the summer of 2012 with the rapid growth 
of enrollments in the three major MOOC platforms. Coursera and Udacity are two for-profit 
Silicon Valley, California start-ups, each led by Stanford University professors. The third, the 
non-profit organization, edX, led by a MIT professor, was initially a partnership between MIT 
and Harvard, but now is a consortium of a number of universities. 


Prior to forming these entities, the faculty involved had experimented with teaching 
their own MOOCs. For example, Sebastian Thrun of Udacity taught introduction to artificial 
intelligence. Andrew Ng of Coursera taught a course in machine learning. Anant Agarwal of 
edX taught a course in circuits and electronics. All of these fully online courses enrolled 
thousands of students from around the world—and in some cases enrollments in an 
individual class exceeded 100,000 students. The enthusiasm behind the development of MOOC 
platforms is closely linked to the personal experiences of the founders. 

The three major MOOC platforms are somewhat distinct from each other in terms of 
mission, strategy, and tactics. At the risk of over simplifying, I will attempt a brief overview. 
With a distinct access mission, Coursera has the largest enrollment with over 3.7 million 
students at the time of this writing. The firm uses a decentralized model, partnering with 
largely elite, “name brand,” universities in the U.S. and globally (though Coursera has 
diversified somewhat) that are mainly responsible for delivering faculty and content. Course 
content leans toward upper division, specialized courses. Coursera provides the platform and 
various instructional and assessment tools, format guidelines, course development support, 
marketing, enrollment, and customer and technical support. 

With a mission of fostering access and successful learning outcomes for students 
currently not well served by higher education, Udacity is the most vertically integrated of the 
three, employing a high degree of instructional design, integrated feedback and assessment 
tools within its courses as well as providing platform, marketing, and student support. Due 
mainly to their detailed and painstaking production methods, Udacity has completed fewer 
courses to date and tends to offer a large proportion of foundational, basic courses, especially 
in math and science areas. 
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The third MOOC platform, edX, is somewhere in between. The nonprofit start-up has 
formed partnerships with universities who provide content; edX also directly contributes to 
course and assessment design, though perhaps to a lesser degree than Udacity. Each platform 
collects a wealth of data on how students are interacting with their courses and the outcomes 
of their efforts. A number of other MOOC platforms have emerged on existing online learning 
management systems (e.g., Blackboard or Canvas). In these cases, the university providing 
the MOOC is responsible for the course design within platform parameters. The MOOCs 
phenomenon is not isolated to the U.S. The Open University in the UK, for example, an 



. RESEARCH if PRACTICE IN ASSESSMENT 

institution with a deep history in distance and online education, has launched its own 
MOOG initiative called Futurelearn. 


Emerging Issues 

MOOGs were initially offered at no cost to students and on a no credit basis. The 
courses were open in the sense they had no prerequisites or admission requirements. 
Many students enrolled to “test the waters” in a new subject area or for their own personal 
enrichment or professional development. The majority of students who enrolled did not 
complete their courses. Within the “no credit context,” course completions are estimated 
to be less than 10% (Agarwala, 2013). However, student motivations for enrolling in MOOGs 
vary and perhaps completion rates do not tell the whole story. Plus, in a course that enrolls 
100,000 students, 10% completion is still a significant number of students. 

We might note the use of the term “open” in MOOG is a bit of a misnomer. For the 
most part, ownership of course content and platform design is asserted and protected by 
course developers, therefore allowing them to monetize their intellectual property in some 
manner. Only edX provides open educational and platform resources in the normal sense of 
“open,” that is material that is freely open and available for use or adaptation by others. 

Earlier this year a number of formal and informal experiments and pilots emerged 
in an attempt to recognize and validate student learning and/or to provide “transcriptable” 
academic credit. For example, anecdotal reports in the media described students who 
completed computer science courses through MOOGs and subsequently listed them on 
their resumes or Linkedin profiles where potential employers might notice them. One 
monetization strategy of for-profit MOOG providers includes acting as an “employment 
agency,” selling information on student performance for students who opt into the service. 

Individual colleges and universities began accepting MOOGs for credit with faculty 
approval or completion of an assessment examination given by the university itself in order 
to receive credit there. The University of Helsinki, Finland, is one institution that employs 
this model (Kurhila, 2012). Other universities licensed MOOC content and integrated that 
into a campus-based course that would be eligible for credit at that particular institution. San 
Jose State University, California, is piloting programs with Udacity and edX MOOGs in such a 
hybrid format (“Frequently Asked Questions,” 2013) as is Antioch University in Los Angeles 
(“Antioch University Becomes the First,” 2013). 


For the most part, however, institutions that sponsor MOOGs do not offer their 
own academic credit to students who complete MOOGs at that institution. In other words, 
currently a non-matriculated student cannot earn University of Pennsylvania credit for 
completing a MOOG offered by Penn. Often a MOOC is qualitatively different from a 
campus course—online format notwithstanding. For courses that are exactly the same 
or equivalent, there appears to be a firewall of sorts between providing academic credit 
for paying, matriculated students at a given institution versus no credit available from 
the institution for the masses of nonpaying students. Protecting the integrity of their full 
residential campus experience appears to play a role. This “firewall” may erode over time. 


MOOCs provide an 
opportunity for the 
assessment community 
to move to the center in 
one of the most exciting 
and potentially important 
conversations emerging 
in higher education today. 


Digital badges are one innovation that has emerged as a means for validating 
student learning whether learning occurs inside or outside the academy. The Mozilla Open 
Badges concept is the leading example and is modeled roughly on the iconic boy or girl 
scout badge. As Knight and Gasilli (2012) elaborate: “A ‘badge’ is a symbol or indicator of 
accomplishment, skill, quality, or interest . . . [that has] been successfully used to set goals, 
motivate behaviors, represent achievements, and communicate success in many contexts” 
(p. 279). Digital badges can be developed by any issuer. Criteria and standards for awarding 
the badge as well as the characteristics and reputation of the issuing organization are made 
transparent under Mozilla’s system (Knight & Casilli, 2012). Badges tend to acknowledge 
narrow and specific skills and competencies and currently are a form of alternative nticro- 
credentialing not linked to formal academic credit as we know it. 
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Ill another experiment, the American Council on Education (ACE) began applying its 
long-standing course review and credit recommendation service to the MOOC environment. 
A pilot review of a group of five courses on the Coursera platform was completed in January 
2013, with all five courses recommended for some form of academic credit. 1 Coursera students 
are now able to opt into the for-credit option in these courses for a fee, usually in the range of 
$100-150 per course. Additional pilots are underway to expand the pool of courses eligible 
for credit recommendations as well as a study to investigate how institutions might apply 
MOOC credit recommendations for their students. Like nearly everything else with this 
educational innovation, the credit issue is evolving, but it likely will remain important in 
the future acceptance of and growth of MOOCs. 2 

Drivers of Change 

MOOCs have focused attention on a number of more general macro trends that 
have tremendous potential to disrupt and change our higher education system. Much of 
the conversation so far has focused on open access provided by MOOCs and the potential 
to educate large numbers of students for a lower per-student cost. It is still debatable whether 
or not MOOCs will fulfill this promise. However, MOOCs have played an important role in 
accelerating discussion on a number of important trends. The following section provides a 
brief overview. 


Attainment goals. The U.S. has long prided itself on having one of the most highly 
educated populations in the world. Unfortunately, that is no longer the case. Over the 
past decade, the proportion of the population with a postsecondary degree has increased 
far more significantly in other advanced economies than in the U.S., particularly among 
young people. According to the Organisation for Economic Cooperation and Development 
(OECD, 2012), only 42% of young Americans ages 25-34 currently hold an associate degree 
or higher. Contrast that to figures for South Korea at 65%, Japan at 57%, and Canada at 56% 
(OECD, 2012). 

Further, 63% of U.S. jobs are projected to require some level of postsecondary 
education by 2018 (Carnevale, Smith, & Strohl, 2010). On a social level, postsecondary 
attainment has always been an important means for providing social equity and 
economic mobility to U.S. citizens. There is a huge gap to fill. We must develop the 
means to provide quality education at a larger scale than ever before. Most national 
attainment goals speak to 60% attainment by 2025. The work ahead of us is daunting 
and MOOCs provide some promise. 


On a social level, post¬ 
secondary attainment has 
always been an important 
means for providing social 
equity and economic 
mobility to U.S. citizens. 


Cost. The cost of higher education to students and families has escalated due 
to reductions in traditional funding sources (Archibald & Feldman, 2012; Desrochers & 
Kirshstein, 2012). Students are graduating with more debt than in the past and default rates 
on student loans are trending upward. MOOCs are currently provided for free or at low cost 
to students. If a student was able to transfer some credit earned by completing MOOCs, this 
would decrease the student’s total cost to complete a degree, certificate or credential similar 
to students who are able to apply transfer credit from Advanced Placement or CLEP exams. 
Though the price currently charged to a student end user enrolled in a MOOC is negligible, 
the cost to produce a MOOC is not. The University of Pennsylvania, an early Coursera 
adopter, estimates its cost to be $50,000 per MOOC, not including faculty time (Popp, 2013). 
MOOCs that incorporate a high degree of design, assessment, and analytics cost much more. 


Globalization. With their global reach and estimates of 60% of enrolled students 
from outside the U.S., MOOCs both reflect and contribute to this trend and illustrate that 
the U.S. higher education system continues to attract students from around the world. 
MOOCs also may offer the potential for domestic students who participate to become 
more globally aware and culturally competent. Attainment goals are frequently linked 
to the need for the U.S. workforce to remain globally competitive (American Council on 
Education, 2011). 
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Competency-centered models. U.S. higher education has been—and still is— 
oriented toward inputs rather than outcomes. If we have the best faculty, students, 
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libraries, facilities, and the right amount of “seat time,” the argument goes, we will have 
optimal student learning outcomes. Most current funding formulas are based on enrollment, 
not graduation. 

Recently there is strong evidence of a shift. Regional accrediting bodies have begun 
to focus attention on outcomes in their accreditation reviews. Some states are beginning 
to integrate performance-based metrics into their funding formulas for public institutions. 
In April 2013, the U.S. Department of Education approved the eligibility of Southern New 
Hampshire University to receive Title IV federal financial aid for students enrolled in their 
new competency-based degree program (Parry, 2013), signaling a distinct willingness to 
move beyond the traditional credit hour measure. 

With its focus on outcomes, the assessment community has inherently questioned 
the validity of the “inputs only” model for some time. None of this conversation is 
completely new. However, with their potential to collect massive amounts of student data 
and to integrate predictive analytics and rapid assessment and feedback systems, MOOGs 
may have played a role in opening up the conversation to more voices and in accelerating 
a reorientation from inputs to outcomes. 


Technology and customization. The constant forward march of technology 
advancement is a consistent trend throughout all aspects of our lives. Within higher 
education and the MOOG environment, technological advancement intersects with cognitive 
learning science and will allow for a higher degree of personalization and customization 
of content and pedagogical methods than ever before. As Soares (2011) points out, the 
Carnegie Mellon Online Learning Initiative has been an early adopter in designing online 
environments that customize content specific to individual student needs. Currently, “smart 
systems” can detect areas where students are having difficulties and then direct students to 
additional resources or practice exercises to help them learn successfully. This capacity will 
only continue to be developed and refined. Analytics and feedback should play an important 
role in contributing to the success for the many students who require additional academic 
preparation in foundational subjects in order to progress toward a degree or credential. 


With its focus on outcomes, 
the assessment community 
has inherently questioned 
the validity of the “inputs 
only” model for some time. 


Open and ubiquitous information. Information is available, open, and free. Yes, 
we need to acknowledge the digital divide in this country—not everyone has access to high 
speed internet. Still, as long as an individual has some access, information abounds. Higher 
education’s traditional role was as repository, organizer, and disseminator of information. 
That role is changing. 


Disaggregation of the faculty role. Related to many of the trends above, we may be 
witnessing an inflection point in how faculty perform their teaching duties. For centuries, 
postsecondarv teaching has been vertically integrated: identifying a subject area, designing 
a course, sourcing content, organizing content, determining learning outcomes, designing 
exercises and assessments, teaching the course, scoring assessments, and assigning final 
grades (Mendenhall, 2012). The need to increase attainment, a shift to a competency- 
centered approach, open access to information, as well as technological advances may focus 
the integrated faculty role to become one of curator of information and mentor to students, 
the components of teaching most valued by faculty. Are we in the position now to explore 
whether student learning outcomes can be improved if course design, technical content 
sourcing, learning technology, and assessment are “outsourced” to experts, leaving sophis¬ 
ticated content curation, course delivery, and personalized student mentoring to those who 
can do that best—faculty? 

Assessment Now Front and Center 


One of the more interesting and promising aspects of MOOCs is the high level of 
experimentation and rapid prototyping of technology-based assessment that has occurred. 
This has very positive implications for assessment scholars and professionals. Because 
of the scale of MOOCs, it would be impossible to hire enough humans to conduct all 
assessments required in a course. Further, the mission of several MOOC providers is to 
improve student learning in foundational courses, especially among first generation, low- 


Volume Eight I Summer 2013 


♦RPA 9 



RESEARCH ir PRACTICE IN ASSESSMENT 


However, with their 
potential to collect 
massive amounts of 
student data and to 
integrate predictive 
analytics and rapid 
assessment and 
feedback systems, 
MOOCs may have 
played a role in opening 
up the conversation 
to more voices and 
in accelerating a 
reorientation from 
inputs to outcomes. 


income students, using adaptive learning and feedback mechanisms. For these reasons, 
assessment methods will be hardwired into a MOOC. 

Standard assessment methods are applied within MOOCs, especially in subjects that 
can be assessed by commonly used objective means. We also are witnessing developments 
in the areas of machine grading and peer grading that can be used to score writing-based 
assessments. Other articles in this issue will address some of these methods in more detail 
(Balfour, 2013). 

The majority of MOOCs offered for credit are in STEM disciplines. It will be interesting 
to see new developments in large scale online assessments for classes in the humanities and 
the arts where multiple choice exam questions are not always the most effective or accepted 
assessment method. 

Early response by faculty to MOOC assessment experiments in machine or peer 
grading shows a relatively high degree of acceptance, even at this early stage. A recent survey 
of MOOC faculty conducted by The Chronicle of Higher Education indicated that 74% of 
respondents used automated grading; 67.1% found the technique to be “very reliable’’ and 
30.1% found it to be “somewhat reliable.” Thirty-four percent (34%) of respondents used peer 
grading; 25.8% found the technique to be “very reliable” and 71% found it to be “somewhat 
reliable” (Kolowich, 2013). 

Predictive analytics and adaptive learning, methods that permit customization of 
content based on student learning, have assessments embedded in them. The sheer volume 
of data being collected on student behavior and learning while interacting with their MOOC 
courses may assist the assessment community in further developing and refining techniques. 
Techniques developed within MOOCs will no doubt migrate into other formats and settings, 
including traditional online and classroom-based courses. 

Related to assessment, and important considerations for those interested in granting 
academic credit for completing MOOC courses are issues of authentication and proctoring. 
In short, the academic community must be confident that the person completing the course 
and assessments is the same person who enrolled in the course. Authentication of identity 
is a common concern in standardized testing and various methods have been employed to 
verify identity, most commonly government issued photo identification, as well as newer 
biometric techniques like palm vein recognition. 

Many of these methods can be converted to the MOOC environment. New methods 
are being developed frequently, like keystroke recognition and queries based on public 
record information (e.g., past addresses) that only an individual would know in detail. 
Authentication might be required for each quiz, assignment, or each time a student logs 
onto the MOOC platform. 

Because of cost, proctoring typically occurs for the summative assessment only and 
is handled in one of two ways. One method requires the student to complete the exam at a 
physical testing center (a public library, educational institution, or private testing facility). 
This requires the student to travel to the testing center site, making it difficult for more 
remote students to participate. 

The second method is webcam proctoring. The student is monitored throughout 
the time of the exam over a webcam. Proctors first scan the room via the camera and 
then ensure that the student is not consulting online or other resources or people for the 
duration of the exam. Authentication and proctoring have associated costs. Currently, these 
services are offered to MOOC students on an optional basis for a nominal fee (usually $ 100— 
$150) and typically would be required for a student to earn academic credit for completing 
the course. Authentication and proctoring are vital elements to provide a high degree of 
confidence in assessments within MOOCs. Expect to see many more technology solutions 
developed in the near future. 
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Final Thoughts 

The rise of MOOGs is an extremely positive development for assessment scholars 
and practitioners. MOOCs have focused our attention and have fostered much excitement, 
experimentation, discussion and debate like nothing I have seen in my decades-long career in 
higher education. MOOGs represent a rapidly evolving landscape. I applaud Research & Practice 
in Assessment (RPA) for diving into the waters to be an early participant in this conversation. I 
expect MOOGs to continue to develop and evolve and I expect what we are learning in the MOOG 
environment to inform and to become integrated in other learning contexts. 

Within the MOOC world, assessment is a central feature of design from the very 
beginning. In this new context, assessment is less about compliance than about supporting 
student learning outcomes and ultimately student success and attainment—directly in the 
center as it should be. 


End Notes 


1 Four courses, Pre-Calculus from the University of California, Irvine; Introduction to Genetics and Evolution from Duke 
University; Bioelectricity: A Quantitative Approach from Duke University; Calculus: Single Variable from the University 
of Pennsylvania, received recommendations for undergraduate credit. Algebra from the University of California, Irvine, 
received a recommendation for mathematics vocational credit. 


2 In full disclosure, the author oversees this program as part of her responsibilities at ACE. The ACE Credit recommendation 
service has existed for over 50 years to assess and assign credit recommendations for formal learning that does not take 
place in a university setting (extra-institutional learning), like military service or corporate workplace education. Teams 
of faculty as well as pedagogical and testing experts review educational activities and provide recommendations on credit 
equivalencies. ACE authorizes production of student transcripts with these credit recommendations that may, at the 
discretion of the degree-granting institution, be applied toward a degree or other academic program. ACE has a network of 
2,000 institutions that regularly consider and accept these credit recommendations. 
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